¹

Thanks, I'm happy to be here, even if I am imaginary.

And please, call me Tipi.

Okay, Tipi.

I can say that I thoroughly enjoyed the book and appreciated the way it carefully developed the key concepts.

How did you find the book?

²

To be honest, I haven't read it all yet. So far I've only skimmed it. It looks interesting, but I'm trying to decide if it is worth the time and effort to work through the whole book.

You have a good point.

This book is more interactive than most, and asks more of the reader.

I can't tell you whether the book is right for you, but I can say that the best way to achieve expertise in any area is with deliberate practice, not just passive listening or reading.

³

It does seem to demand more effort than a typical book, because you have to work through the examples.

Working through the examples really helps the material stick with you!

I remember one time I was discussing a machine learning technique with a colleague, and they said they didn't see how a particular tricky part worked. I said that it seems tricky, but it is actually easy.

I remembered that Andrew Ng had a great video that was perfectly elegant in explaining how it worked. I then tried to duplicate what I learned from the video and—I couldn't do it.

Andrew's explanation was so clear when I watched it that I didn't bother actually learning it. My colleague did a quick search and said ‘‘This must be the video; let's watch it again,’’ but I said no; this time I'm going to work the answer for myself, and then I'll remember it.

And I do remember it to this day.

⁴

Okay, you convinced me that practice makes improvement.

When I look around, however, I see machine learning tools mainly in Python, or maybe Java or R, but not so much in Scheme.

This book isn't about programming languages, and it isn't about machine learning toolkits.

It is about explaining the fundamental concepts of machine learning, and implementing them in the simplest possible way.

⁵

Is Scheme a good choice for that?

I think it is.

It is one of the simplest possible programming languages. Working with it is like sticking with basic mathematical notation, but it happens to be executable code, so there's no ambiguity about how it all works.

⁶

But will this book have me practicing the right things?

After you finish this book, are you going to take the Scheme code and run it in your next machine learning project?

Maybe, maybe not. But even if you use a machine learning toolkit like TensorFlow or PyTorch, what you will take away from this book is an appreciation for how the fundamentals work.

That appreciation will help you understand what to try next when TensorFlow or PyTorch is not behaving the way you thought it should.

⁷

Okay, I'm all for understanding the fundamentals. But is this book all theoretical? All mathy? I want a good understanding of the fundamentals, but I want that understanding to be useful in practice.

This book will give you that. It takes you through the key ideas of machine learning.

It is definitely not “mathy’’; it has the minimum amount of math notation, and most topics are described with code and prose, not equations.

Personally, I'm the kind of person who is always more comfortable reading code than math, so this book filled me with cozy feelings, not with dread.

⁸

That sounds good. What are some of the key topics I will learn about? And how long does it take to get to them?

I think the topics are presented at just the right pace. It starts off introducing the absolute basics of Scheme and of lines and linear equations. By page 63 we get to L2 loss, learning rate, and gradient descent.

Often the topics are covered in a simple way first, and then in a full-blown practical way later; we come back to gradient descent on page 141 and get to Adam optimization (a state-of-the-art version of gradient descent) by page 173.

By the time we get to page 201, The Rule of Artificial Neurons, “An artificial neuron is a parameterized linear function composed with a nonlinear decider function’’ makes perfect sense, putting together the pieces that have been built up in the preceding pages.

We learn about the most commonly-used decider functions, like the “relu,’’ and on page 209 there's a simple but deeply satisfying description of how any function can be approximated with the tools we have at hand.

The book continually builds on solid foundations like this to explore the issues that come up in practice; for example in the discussion of vanishing and exploding gradients on page 260.

⁹

Okay, you've convinced me; skimming wasn't enough, I'm going to work through the whole book.

By the way, was there anything in the book you disagree with?

I whole-heartedly agree with the pedagogical style of working through an issue, seeing the finished results that you have accomplished, and then enjoying a break.

I've used the Pomodoro Technique to help keep me focused on work intervals; this book is perfectly suited for such an approach.

I think, however, if you strictly followed the book's advice for the number of desserts to eat, you'd be overdoing it.

Take the functions, leave the cannoli!

¹⁰

As a typical reader, I consume 77 grams of sugar per day.

I'll heed your warning and try to get closer to the daily recommended level of 24 grams per day.

24:17

[ t ts …]

(tensor t ts …)

26:24

l_i

(ref l i)

27:27

(list m …)

(list m …)

33:17

(tlen t)

36:24

t|_i

(tref t i)

41:42

|l|

(len l)

52:22

(⟨op⟩⟨rank⟩ t)

(⟨op⟩-⟨rank⟩ t)

77:15

(∇ f θ)

(gradient-of f θ)

106:25

(• t u)

(dot-product t u)

106:26

(⟨op⟩⟨rank₁⟩,⟨rank₂⟩ t)

(⟨op⟩-⟨rank₁⟩-⟨rank₂⟩ t)

124:27

t||_b

(trefs t b)

226:49

li↓

(refr l i)

¹

It's good to be here!

Indeed.

Remember this?

²

Yes.

That looks like a line in plane (2-dimensional) geometry along with two additional lines, the x-axis and the y-axis.^†

There is an equation that relates x with y for every point (x, y) on a line

³

Does using arrows at both ends of the line mean that it extends indefinitely in both directions and does it follow that there is a corresponding y for every x?

Yes.

Our line passes through the origin.

⁴

What is the origin?

The point at which the x-axis and the y-axis meet is known as the origin and is the point (0, 0). The dark line in the picture passes through the origin.

⁵

So what is the equation of this line?

Because this line passes through the origin, y is a multiple of x by a constant factor w, which is known as the slope of the line

⁶

Does that mean y = wx?

Yes, it does.

⁷

What if the line does not go through the origin?

Good question. This is what it would look like

How can we determine y?

⁸

It appears that the whole line is lifted by b now.

We can determine y for any x using

y = wx + b^†

Here is our first attempt at line^†

⁹

This is a function of one argument that results in a function of two arguments?^‡

This dashed definition of line seems backwards. Why don't we require w and b before x?

Great question!

That kind of definition assumes that w and b are known prior to the argument x.

Here, though, we deal with a different kind of problem, where x is known, but w and b must be figured out from a bunch of given values of x and y.

¹⁰

Why isn't line final?

That's because our next attempt can simplify line a little more. Since the body of the let, y, is only the name given to

(+ (* w x) b)

by the let, we get this same-as chart^†

1. 1.  (let ((y (+ (* w x) b)))
2.      y)
3. 2.  (+ (* w x) b)

Even though there is no longer a y here, we sometimes refer to this value as the y associated with a given x.

¹¹

So, this dashed definition should be our next attempt to finalize line

It looks correct, so why is line still dashed?

Good question.

This line is still dashed because we are going to make another attempt soon. But for now, it suffices.

Because w and b are used to determine the y corresponding to a given x, they are considered to be a special kind of formal. We name them parameters of line, and we use bold letters for them, whereas x is the argument of line.

¹²

How is the function line used?

Let's see an example.

What is

(line 8)?

¹³

(line 8) is a function that remembers that x is 8, and is waiting to accept arguments for its parameters w and b

(λ (w b)

 (+ (* w 8) b))

That is correct.

When (line 8) is invoked on w and b, we can determine y.

What is

((line 8) 4 6)?

¹⁴

Here are the steps with w being 4 and b being 6

1. 1.  ((line 8) 4 6)
2. 2.  ((λ (w b)
3.      (+ (* w 8) b))
4.     4 6)
5. 3.  (+ (* 4 8) 6)
6. 4.  (+ 32 6)
7. 5.  38

This means that when x is 8, y is 38.

Excellent.

Functions that accept parameters after the arguments are known as

parameterized functions

Is line a parameterized function?

¹⁸

Yes.

It takes w and b as its parameters after it takes the argument x.

Why are parameterized functions special?

Good question.

Parameterized functions are used where we must figure out the right values for the parameters (here, w and b) from given values of x and the corresponding values of y.

¹⁹

Could we see an example?

Sure.

Here line-xs are the x-coordinates and line-ys are the y-coordinates

For each x-coordinate in line-xs, there is a corresponding y-coordinate in line-ys and vice versa.

¹⁷

Do line-xs and line-ys, when taken together, give us these four points?

• (2.0, 1.8)

• (1.0, 1.2)

• (4.0, 4.2)

• (3.0, 3.3)

Yes.

Together, they form a data set.

¹⁸

Okay.

Here is how the data set

(line-xs, line-ys)

looks as points on a graph

¹⁹

Those points look like they are almost on a straight line.

For good reason.

Draw a line that is close enough to these four points.

²⁰

How about this?

Yes, that is close enough.

Lines, as in frame 11, have two parameters, w and b.

What are w and b for this line?

²¹

Since this line passes through the origin, b is 0.0.

And, the x and y coordinates of every point on this line are always equal. For example, (0.0, 0.0), (1.2, 1.2), etc. The points on this line look like (a, a), for any x-coordinate a.

So, w must be 1.0.

That is correct.

This function is simply a line with parameters

w = 1.0

and

b = 0.0

How can we use this information?

²²

If we are given a new x-coordinate, we can predict the corresponding y-coordinate from this line.

For example, if x = 3.79, then

y = 1.0 × 3.79 + 0.0 = 3.79

²³

So we have learned a function that behaves approximately like the points given by the data set

(line-xs, line-ys)

The parameters w and b here are collectively known as the parameter set, which we refer to as θ.^†

So, our θ has two parameters. They are referred to as members of θ.

Then w is the first member of θ, which we write as θ₀.^‡

²⁴

Is b then θ₁?

It is.

Let's rewrite line to reflect this

²⁵

Why is this line in a red-colored box with a red-colored circle?

Good question.

We have chosen red boxes and circles to highlight final definitions that are important as well as sometimes being primitive.^†

²⁶

Red is such a vibrant color for a toy. How do we construct a θ for w and b?

Great question.

Here's an example. If w is 1.0 and b is 0.0, we construct it this way

(list 1.0 0.0)

²⁷

So θ is a list of members such that

θ₀

which is the same as w is

1.0

and

θ₁

which is the same as b is

0.0

Here's an example of how line is invoked with this θ

1. 1.  ((line 7.3) (list 1.0 0.0))
2. 2.  ((λ (θ)
3.      (+ (* θ₀ 7.3) θ₁))
4.     (list 1.0 0.0))

Complete this same-as chart.

²⁸

Sure

1. 3.  (+ (* (list 1.0 0.0)₀ 7.3) (list 1.0 0.0)₁)
2. 4.  (+ (* 1.0 7.3) 0.0)
3. 5.  7.3

Correct.

So far, we have examined our data set on a graph and estimated a θ.

²⁹

And we used it to predict a

y-coordinate

for an

x-coordinate

that may not be in our data set.

In other words, we have learned from an existing data set how to make y-coordinate predictions for x-coordinates.

³⁰

Yes, we have learned something!

Still, we have accomplished this by visually inspecting only the points on the graph.

³¹

Can we define a function that determines θ for any data set?

An excellent question.

Yes, we can, and such a function is an example of what is known as machine learning.^† But that's what the rest of the book is about!

³²

It's a good time to take a break.

¹

Mmm . . . fantastic!

Is 5 a natural number?

²

Yes.

It is a nonnegative integer.

What about 0?

³

Yes.

It is a natural number.

Is −5 a natural number?

⁴

No.

Natural numbers cannot be negative.

What is 7.18?

⁵

That's a real number.

Is −13.713 also real?

⁶

Yes, it is.

Are all natural numbers also real numbers?

Yes, we consider them to be.

There is another name we use for all real numbers here. We refer to them as

scalars

⁷

Is 7.18 a scalar?

Yes.

What is π^†?

⁸

Like 3.141592653589793 . . .? Then, π is also a scalar.

Correct.

The predicate scalar? tests whether something is a scalar and like all predicates is a function that always results in a Boolean—either false #f or true #t.^†

⁹

Okay.

Here is a tensor¹.^† A tensor¹ has only scalars

[5.0 7.18 π]

¹⁰

Is that a superscript on the word tensor^‡?

Yes, it is.

A tensor¹ groups one or more scalars together^†

[2.0 1.0 4.0 3.0]

¹¹

These bracketed scalars seem familiar!

They should be, that's line-xs from frame 24:17.

Here is another example

[8]

¹²

This tensor¹ contains the scalar 8.

Is there a tensor^2†?

Yes, the elements of a tensor² are tensors¹.^†

¹³

What is an element?

For example

[[7 6 2 5] [3 8 6 9] [9 4 8 5]]

has 3 elements

[7 6 2 5]

[3 8 6 9]

and

[9 4 8 5]

¹⁴

Then if we have a tensor whose elements are tensors^m, does that make it a tensor^m+1?

Yes, but there's a condition.

All the tensors^m must have the same number of elements.

¹⁵

That seems reasonable.

What can we say about the 3 tensors¹ in frame 14?

¹⁶

Each has 4 elements and each of those elements is a scalar.

The number of elements in a tensor t can be found by invoking (from page xxiii)

For example

1. 1.  
2. 2.  4

How about

¹⁷

This is it

1. 1.  
2. 2.  2

Yes, it is.

¹⁸

Is this

[[[[8]]]]

also a tensor?

Yes, the tensor

[[[[8]]]]

is a tensor⁴ of an element

[[[8]]]

which is a tensor³ of an element

[[8]]

which is a tensor² of an element

[8]

which is a tensor¹ of an element

8

which is a scalar.

What is

[[5 6 4] [9 1 1] [0 6 2]]?

¹⁹

It looks like a tensor² whose three elements are tensors¹.

Is this

[[[5] [6] [7]] [[8] [9] [0]]]

possible?

Yes, the tensor

[[[5] [6] [7]] [[8] [9] [0]]]

is a tensor³ of

2 tensor² elements

each of those 2 elements has

3 tensor¹ elements

each of those 3 elements has

1 single element

a scalar.

²⁰

Since a tensor^m+1 must have only tensors^m elements, a tensor¹ should be made up of tensor⁰ elements.

Does that mean a scalar is a tensor⁰?

Great question.

Yes, a scalar like 9 is a tensor⁰.^†

²¹

Does the superscript have a name?

Yes, it does.

It is known as the rank of the tensor.

²²

What does the rank of a tensor mean?

The rank of a tensor tells us how deeply nested its elements are.

Here is a tensor³ because it has 1 tensor² element that has 2 tensor¹ elements of two scalars each

[[[8 9] [4 7]]]^†

²³

It appears we can define a function to determine the rank of a tensor.

²⁴

How do we use this to find the rank of a tensor?

Here is a function that finds the rank of a tensor

Here, t|₀ is the 0th element of t, which always exists because we have already determined that t is not a scalar, and furthermore the 0th element is itself a tensor.

Now explain rank.

²⁵

If the base test,^‡ (scalar? t), succeeds then we know the rank of the tensor is 0.

In general, if the elements of a tensor are tensors^m, then the tensor has rank

m + 1

So, in the recursive case, we find

m

which is the rank of the 0th element of t

(rank t|₀)

and then add1 to the result to get the rank of t.

Could we see how rank is 3 with the tensor in frame 23 using a same-as chart?

Sure.

Given the same tensor from frame 23 [[[8] [9]] [[4] [7]]] here is a same-as chart that shows how its rank is 3

1. 1.  (rank [[[8] [9]] [[4] [7]]])
2. 2.  (add1 (rank [[8] [9]]))
3. 3.  (add1 (add1 (rank [8])))
4. 4.  (add1 (add1 (add1 (rank 8))))
5. 5.  (add1 (add1 (add1 0)))
6. 6.  (add1 (add1 1))
7. 7.  (add1 2)
8. 8.  3

²⁶

That is so elegant and natural.

There's another observation here. We need look only at

t|₀

i.e., only the first element of the tensor.

²⁷

Yes, why is that?

That is because in any given tensor, the nested tensors have the same number of elements. For example, the nested tensors of tensors² are all tensors¹, and each of those tensors¹ has the same number of tensors⁰.

²⁸

That's the requirement from frame 15.

Correct.

This means that the tensors^m that are elements of a tensor^m+1 have the same shape.

²⁹

What is the shape of a tensor?

The shape of

[[5.2 6.3 8.0] [6.9 7.1 0.5]]

is this list of positive natural numbers

(list 2 3)^†

Where do those natural numbers come from?

³⁰

This tensor is a tensor² of 2 tensors¹, each of which has 3 tensor⁰ elements.

Exactly.

What is the shape of

[[[5] [6] [8]] [[7] [9] [5]]]?

³¹

The shape of

[[[5] [6] [8]] [[7] [9] [5]]]

is

(list 2 3 1)

Right again.

[[[5] [6] [8]] [[7] [9] [5]]]

is a tensor³ of

2 tensor² elements

Each of those has

3 tensor¹ elements

Each of those has

1 tensor⁰ element

which is a scalar.

What is the shape of

[9 4 7 8 0 1]?

³²

[9 4 7 8 0 1] is a tensor¹ of 6 scalars.

So, is

(list 6)

the shape of

[9 4 7 8 0 1]?

That is correct.

³³

Do scalars have a shape?

They do.

Since scalars have no parts, their shape is the empty list.

³⁴

How do we write empty lists?

Like this

(list)

³⁵

So the shape of the scalar 9 is

(list)?

Correct.

To find the shape of a tensor t, we need to know the number of elements it has.

³⁶

So we use for it?

We do!

Here is the function shape that finds the shape of a given tensor t

Explain this shape.

³⁷

But wait, what is cons?^‡

³⁸

When the argument to shape, t, is a scalar, its shape is simply the empty list.

Otherwise, it is a non-scalar tensor, so we find the 0th element's shape, which is a list, by recursively invoking shape thusly

(shape t|₀)

The shape of t is then consed to the front of the resultant list.

Correct.

What is

(shape 9)?

³⁹

(shape 9) is the empty list because 9 is a scalar.

What is

(shape [9 4 7 8 0 1])?

⁴⁰

(shape [9 4 7 8 0 1]) is

(cons 6 (shape 9))

which is

(list 6)

⁴¹

Here it is

1. 1.  (shape [[[5] [6] [8]] [[7] [9] [5]]])
2. 2.  (cons 2 (shape [[5] [6] [8]]))
3. 3.  (cons 2 (cons 3 (shape [5])))
4. 4.  (cons 2 (cons 3 (cons 1 (shape 5))))
5. 5.  (cons 2 (cons 3 (cons 1 (list))))
6. 6.  (cons 2 (cons 3 (list 1)))
7. 7.  (cons 2 (list 3 1))
8. 8.  (list 2 3 1)

The number of members in a list ls is

|ls|

How are rank and shape related?

⁴²

Aha!

The number of members in the shape of a tensor^† is also the rank of the tensor

(= |(shape t)| (rank t))

⁴³

Can't wait.

Here's our final way to define rank, but this one is already unwrapped, i.e., no recursive invocation is wrapped!

This rank definition uses a support function ranked that includes an additional formal a as an accumulator.

To see how it works, repeat the example in frame 26 but using this final rank.

⁴⁴

Here it is

1. 1.  (rank [[[8] [9]] [[4] [7]]])
2. 2.  (ranked [[[8] [9]] [[4] [7]]] 0)
3. 3.  (ranked [[8] [9]] (add1 0))
4. 4.  (ranked [[8] [9]] 1)
5. 5.  (ranked [8] (add1 1))
6. 6.  (ranked [8] 2)
7. 7.  (ranked 8 (add1 2))
8. 8.  (ranked 8 3)
9. 9.  3

Instead of a wrapper around the recursive invocation of

rank

we

add1

to the accumulator

a

as we are going down into

t|₀

That is correct.

⁴⁵

Thanks!

Seeing this way of using t|₀ as an argument in an unwrapped recursive invocation helps.

Unlike the dashed rank in frame 25, this new recursive function definition does not use a wrapper.

In fact, this matters so much that we have a law about it.

⁴⁶

Let's see it.

⁴⁷

Why?

This law enables us to handle very large tensors and lists.

⁴⁸

How?

When combining simple accumulator passing function definitions, they could be thought of as one very big loop.

⁴⁹

How is that possible?

Every Scheme system is required to support tail call optimization which makes each unwrapped recursive invocation behave the same as a loop.

⁵⁰

How do the other requirements in the law help?

The other requirements ensure that we use a uniform pattern for our function definitions so that they are easy to read and understand.

⁵¹

These simple accumulator passing function definitions seem to offer great possibilities!

Now it is time for a break!

⁵²

It is just what is needed. Things were getting intense.

¹

Boba with mango jelly, scrumptious!

How about an interlude?

²

What's an interlude?

It's where we temporarily shift our focus.

³

To what?

To have more fun with +.

⁴

That's addition, right?

Yes.

What is (+ 1 1)?

⁵

That's easy

(+ 1 1)

is

2^†

Thank goodness.

What is

(+ [2] [7])?

⁶

That's tricky. The arguments are tensors¹.

Is

(+ [2] [7])

the same as

[9]?

Yes, but why is that?

Here is a same-as chart to discover how the chart results in [9]

1. 1.  (+ [2] [7])
2. 2.  [(+ 2 7)]
3. 3.  [9]

⁷

Why are those brackets turquoise?

When there is a function invocation like + on tensors, we use turquoise brackets to emphasize that we're going to look inside those tensors to help determine the invocation's final value.

What is

(+ [5 6 7] [2 0 1])?

⁸

Is

(+ [5 6 7] [2 0 1])

the same as

[7 6 8]?

Here is the same-as chart that shows how to get that result

1. 1.  (+ [5 6 7] [2 0 1])
2. 2.  [(+ 5 2) (+ 6 0) (+ 7 1)]
3. 3.  [7 6 8]

⁹

It appears that + descends into its

tensor¹ arguments

to result in another tensor¹.

The last step results in the tensor¹ of the values of the three sums.

Yes.

What is

¹⁰

We're adding two tensors² of the same shape. It is

Must we have tensors of the same shape before we can add them?

Yes, we must.

Getting + to work on tensors of arbitrary rank is known as

the extension^† of +.

Functions built using extension are known as extended functions. Can other functions that work on scalars be extended similarly?

¹¹

It would appear so.

There is nothing special about +. Other extended scalar functions should work in the same way.

Here's another way extension works.

What is

(+ 4 [3 6 5])?

¹²

But these two tensors don't have the same shape!

Correct.

When that happens, we do this

1. 1.  (+ 4 [3 6 5])
2. 2.  [(+ 4 3) (+ 4 6) (+ 4 5)]

Finish this same-as chart.

¹³

Oh, we look inside the tensor¹ argument and repeatedly add 4 to each element.

The final answer is

1. 3.  [7 10 9]

Very good.

How about this?

1. 1.  (+ [6 9 1] [[4 3 8] [7 4 7]])

¹⁴

We can look inside the tensor² argument and add the tensor¹ argument to each element, just as we did in frame 13

Excellent.

Let us now take an

extended version of *

This is The Hadamard ^† multiplication.

Now this one

1. 1.  (* [[4 6 5] [6 9 7]] 3)

¹⁵

Here are the steps

The first two steps use turquoise brackets because we are descending into those tensors¹. The last two steps don't have turquoise brackets because we have only scalars remaining, so we just multiply them without descending.

Let's look at another extended function

sqrt^†

1. 1.  (sqrt 9)
2. 2.  3

¹⁶

Applying sqrt to the scalar 9 gets its square root, which is 3.

Now let's invoke sqrt on

[9 16 25]

This time the rank of the argument is 1. So sqrt descends into the tensor¹

1. 1.  (sqrt [9 16 25])
2. 2.  [(sqrt 9) (sqrt 16) (sqrt 25)]
3. 3.  [3 4 5]

¹⁷

The sqrt function has been invoked on each scalar in the tensor¹.

In these same-as charts, again, the tensor that is to be descended into is marked with turquoise brackets.

¹⁸

Could we see another example?

Sure.

Here is an example of how sqrt behaves on a tensor²

¹⁹

It seems as if sqrt descends into each tensor¹ of the argument until it finds the tensors⁰ and gets their square roots.

Yes, descending into a tensor is a trick we use often.

With a higher-rank tensor, the process would repeat itself until we encounter scalars and get their square roots.

²⁰

That makes sense.

Can we treat functions of two arguments in a similar way?

Yes, we can.

We now know how extended functions of two arguments can be made to work when the two arguments have different ranks by descending into the higher-ranked tensor.

²¹

Do extended functions always descend until they find scalars?

Some don't.

To see that, let's look at a new function

sum¹

And this is how we expect it to behave

1. 1.  (sum¹ [10.0 12.0 14.0])
2. 2.  36.0

²²

It looks as if it is summing the scalars in a tensor¹.

But why do we have a superscript 1 on sum?

The superscript is a reminder that sum¹ expects a tensor¹.^†

Now that we know that sum¹ always takes a tensor¹, let's define it.

²³

That sounds interesting.

Here is sum¹

Explain sum¹.

²⁴

We invoke the support function summed with the tensor

t

the last index in t

(sub1 )

and an accumulator starting at

0.0

The function summed counts the index i down to zero. At each step it adds the ith element in the tensor t to the accumulator, and recursively invokes summed.

And finally, when we reach the 0th element in the tensor, we add that element to the accumulator which results in the sum of the entire tensor.

We refer to the extended version of sum¹ as sum, which descends into its argument until it finds a tensor¹ instead of a tensor⁰.

²⁵

How does sum work?

Here is sum working on a tensor³

What can we say about the rank of this result?

²⁶

This is a tensor², so its rank is 2, which means that the rank is 1 less than the rank of the input, which is also true for sum¹.

²⁷

What about functions that are constructed from these extended functions?

A great question.

If we use extended functions to form new functions, then the new functions also automatically work in an extended fashion.

²⁸

Could we see an example of this?

Sure.

Here is such an example.

What is

((line [2 7 5 11]) (list 4 6))?

²⁹

We're invoking line on a tensor¹ argument.

Does that mean the functions + and * in line are extended functions?

It does!

Here, we want to find y values for many (here, four) x values on the line with θ₀ being 4 and θ₁ being 6.

³⁰

Magnificent!

Finish the same-as chart to determine the corresponding four predicted ys

1. 1.  ((line [2 7 5 11]) (list 4 6))

³¹

Here goes

1. 2.  ((λ (θ)
2.       (+ (* θ₀ [2 7 5 11]) θ₁))
3.      (list 4 6))
4. 3.  (+ (* 4 [2 7 5 11]) 6)
5. 4.  (+ [(* 4 2) (* 4 7) (* 4 5) (* 4 11)] 6)
6. 5.  (+ [8 28 20 44] 6)
7. 6.  [(+ 8 6) (+ 28 6) (+ 20 6) (+ 44 6)]
8. 7.  [14 34 26 50]

This interlude is about how it is possible to extend functions that operate on fixed ranks of tensors into functions that also accept tensors of different ranks. We use the concept of descending into the higher-ranked tensors to accomplish this.

³²

And we have learned that by doing this, we descend into the higher-ranked tensor while repeating the lower-ranked tensor until the one- or two-argument operation can proceed.

¹

The beignets went great with it.^†

Let us now return to defining a function to automatically find the θ that best fits (from frame 27:29) the given data set in 24:17. We refer to this θ as well fitted.

²

Just as we visually fit a line to match our points from frame 25:20?

Exactly.

Here we introduce

successive approximations^†

³

Are successive approximations a way to find a well-fitted θ?

Yes.

We determine θ₀ and θ₁ by arbitrarily starting them at 0.0. We then repeatedly revise θ to bring it as close as we wish to what it must finally be.^†

⁴

An example, please.

Sure.

Here is the data set from frame 24:17

line-xs is [2.0 1.0 4.0 3.0]

and

line-ys is [1.8 1.2 4.2 3.3]

⁵

So then do we start with

θ₀ = 0.0

and

θ₁ = 0.0?

That is a fine place to start.

What line does this θ give us?

⁶

We can find out by using (list 0.0 0.0) as θ

1. 1.  ((line line-xs) θ)
2. 2.  ((line [2.0 1.0 4.0 3.0]) (list 0.0 0.0))
3. 3.  (+ (* 0.0 [2.0 1.0 4.0 3.0]) 0.0)
4. 4.  [0.0 0.0 0.0 0.0]

The y-coordinates produced for the x-coordinates in line-xs are all 0.0. So this θ represents the x-axis!

Indeed it does.

Those values, which we refer to as the predicted ys from frame 26:23, are very different from the given line-ys!

⁷

What does this mean?

It means that θ₀ and θ₁ do not yet fit our data set.

⁸

Do we have to revise θ₀ and θ₁ to bring the predicted ys closer to the given line-ys?

Yes, we do.

But, in order to do that, we need to know how far away we are from the well-fitted θ.

We need a single scalar that tells us how close or how far away we are. This scalar is known as the loss.^† The well-fitted θ would be the one where this loss is as close to 0.0 as possible.

When the loss is 0.0, it is ideal.

⁹

Why can't it be a tensor¹?

With a scalar, it is simpler to decide how to revise θ.

¹⁰

How is this scalar used?

We determine this scalar, or loss, every time we revise θ. Since the loss shows us how far away we are from the well-fitted θ, we use it as a guide for revising θ.

¹¹

How do we find the loss for a given θ ?

We'll define a function for it, of course!

But we'll do it in two steps. First we determine a tensor representing how far away we are and then reduce this tensor down to a scalar.

The simplest way to determine how far away we are is to find the difference between the given line-ys and the predicted ys as we see in this skeleton

Find P.

¹²

P must be the predicted ys. So, P must be

((line line-xs) (list θ₀ θ₁))

Correct.

This gives us

(− line-ys ((line line-xs) (list θ₀ θ₁)))

It, however, is not yet a scalar. It is still a tensor¹, so we must turn it into a scalar.

What function, which we have already defined, derives a scalar from a tensor¹?

¹³

That function is sum from frame 53:25.

Is this

(sum

  (− line-ys ((line line-xs) (list θ₀ θ₁))))

how we find the scalar?

Almost, but it is not good enough.

Here's why. Suppose we had another data set and another θ that would have given us a difference of

[4.0 −3.0 0.0 −4.0 3.0]

What would be the result of

(sum

  [4.0 −3.0 0.0 −4.0 3.0])?

¹⁴

It would be 0.0, the ideal loss, which it clearly is not!

So, even though the individual differences are very high, the sum of 0.0 suggests that the θ is a perfect fit.

Aha! This problem arises from having negative values in the argument to sum.

So what should we do?

We solve this by squaring each element.^† This turns the negative elements of the difference to positive, and the positive elements stay positive

(sum

  (sqr

   (− line-ys ((line line-xs) (list θ₀ θ₁)))))

Here, the (extended −) descends into the two tensors¹ resulting in a tensor¹ of differences, then sqr squares the resultant tensor¹ of differences, yielding a tensor¹ for sum to produce a scalar.

Now the sum of the squares is positive if at least one of the elements in the difference is non-zero.

¹⁵

That's a promising strategy.

How should we define our loss function?

Here is a function that does this

¹⁶

Why is this function named l2-loss?

It is an instance of a family of functions that use exponents to find the loss. Here, since we use sqr (i.e., raised to the power of 2), this loss function is l2-loss.^†

¹⁷

Why is this definition dashed?

Because we're going to generalize l2-loss.

This l2-loss relies on line. But, as far as this l2-loss is concerned, line is just some function that results in a predicted ys, pred-ys, when given xs and θ.

¹⁸

How can we use this observation?

We can make l2-loss less specific to line, so we can use it with other functions as well. We redefine l2-loss to take line as an argument

¹⁹

Why is this l2-loss dashed?

This l2-loss is dashed because the name line is too specific.

Since we have made line into a formal, the name no longer represents the original line function. So, we should rename the formal line to something more meaningful.

²⁰

So what name should we pick?

Here, the function line is referred to as a target function because that is the function whose θ we are trying to find for a given data set.

So, instead of the name line, let's use the name target.^†

²¹

Sounds better.

Here is how we define the final l2-loss to use the name target instead of line

²²

Aha!

The dashed l2-loss in frame 19 before we generalized it is specific to the function line. If we want to use it for the function line again, how can we use this generalized form of l2-loss?

²³

We should invoke l2-loss with line

(l2-loss line)

Correct.

Here's the same-as chart for this

1. 1.  (l2-loss line)
2. 2.  (λ (xs ys)
3.      (λ (θ)
4.       (let ((pred-ys ((line xs) θ))
5.         (sum
6.         (sqr
7.          (− ys pred-ys))))))

This function, which is produced when l2-loss is invoked with a target function, is referred to as an expectant function.

²⁴

Why do we refer to

(λ (xs ys) . . .)

as an

expectant function?

That's because it is expecting a data set as arguments.

²⁵

That makes sense.

What does an expectant function produce when it receives a data set?

Let's find out

1. 1.  ((l2-loss line) line-xs line-ys)
2.     (λ (θ)
3.      (let ((pred-ys ((line line-xs) θ)))
4.       (sum
5.        (sqr
6.         (− line-ys pred-ys)))))

This function, which awaits a θ as its argument, is known as an

objective function

²⁶

What does the objective function do?

When provided with a θ, the objective function produces a scalar representing the loss, which is a measure of how far away we are from the well-fitted θ.

²⁷

So, it is going to help us achieve our objective of finding a well-fitted θ!

Let's try it with our current θ₀ and θ₁ which are still 0.0

(((l2-loss line) line-xs line-ys)

  (list 0.0 0.0))

Looking at the final l2-loss in frame 22, we must first determine pred-ys, which is

((target xs) θ)

Use the ideas in the same-as chart from frame 55:31 to help find pred-ys for line-xs and our θ.

²⁸

Since the value of target here is the function line, pred-ys is determined by

Great.

Keeping this value of pred-ys in mind, we find the loss for this θ

Complete this same-as chart.

²⁹

Here it is

The loss tells us how far away we are and 33.21 tells us we are quite far away.

How do we revise our θ to get the scalar loss 33.21 closer to the ideal loss, 0.0?

We begin by testing the behavior of θ₀ to see how we should revise it; we'll worry about θ₁ later.

We change θ₀ by increasing it by a small amount, for testing purposes, so that

θ₀ = 0.0099

Find the new pred-ys for this new θ.

³⁰

Okay

Perfect.

Here is the same-as chart to find the new loss for this new θ₀

Now, complete this same-as chart.

³¹

Here it is

The loss has gone down.

In other words, we are slightly closer to the ideal loss!

We changed the loss by

(32.59 − 33.21) = −0.62

by revising θ₀ from 0.0 to 0.0099.

Our test has succeeded.

³²

So should we continue to revise θ₀ in increments of 0.0099 until we come as close as possible to the ideal loss?

It could mean that.

Revising θ₀ every time by 0.0099 may require too many revisions to get to the well-fitted θ.

There is a way, however, that takes fewer revisions.

³³

Show us the way!

Remember that increasing θ₀ by 0.0099 has changed our loss by −0.62. So our rate of change^† is

³⁴

How do we use this rate of change?

The rate of change of a function (here, of the objective function), determines how its result changes when its argument (i.e., θ) is revised.

By using the rate of change cautiously, we can get to the well-fitted θ with fewer revisions.

³⁵

What does using it cautiously mean?

This rate of change has a large absolute value.^† This means that a small increase in θ₀ causes a relatively large decrease in its loss.

We can use this idea to determine how much to further revise θ₀ so that we can make an even bigger reduction in loss. But, we must be careful.

³⁶

What should we be wary of?

We have to be wary that our revision of θ₀ always moves us closer to, but does not overshoot, the ideal loss.

One choice, for example, is to revise θ₀ by 62.63, which is the absolute value of the whole rate of change. When we do this, however, we end up with a

loss of 113763.027

This is far bigger than 32.59, which is our previous loss, and much greater than the ideal loss.

³⁷

That is something to be wary of indeed.

How do we solve this problem?

We take a small scalar (like 0.01), and multiply the rate of change by it, and revise θ₀ by that amount.

³⁸

So we get

0.01 × −62.63 = −0.6263

Is this a small enough revision that won't overshoot the ideal loss?

Yes, it should be.

This small scalar is known as the

learning rate

Since we use this scalar often, we have a special symbol for it

α^†

We rewrite our example thusly

α × −62.63 = −0.6263

³⁹

Okay.

Since we need to increase θ₀ to reduce the loss, we must subtract this negative value from our current θ₀. So the new θ₀ is

0.0 − −0.6263 = 0.6263.

⁴⁰

But, didn't we set θ₀ to 0.0099?

That was just to find the rate of change.

Once we find the rate of change, we forget about 0.0099.

⁴¹

Okay.

So we revise by multiplying the learning rate and the rate of change and then subtracting the result from our current

θ₀?

⁴²

We use this new pred-ys to find the loss

We have gone from a loss of 33.21 to a loss of 5.52. That is so much better than just increasing θ₀ by 0.0099 as we did in frame 30!

Amazing. We've taken a big step closer to the ideal loss now.

We repeat this process to get our next revision.

⁴³

Can we subtract −0.6263 from θ₀ again?

No, we cannot!

Let's find out what the rate of change of loss is with θ₀ at 0.6263. Let us give

((l2-loss line) line-xs line-ys)

the temporary name obj. Then the rate of change is

⁴⁴

It's different from the rate of change we found in frame 34.

Does this mean that the rate of change depends on the current θ₀?

That is exactly what it means.

⁴⁵

So do we determine the rate of change again using θ₀ + 0.0099?

We could.

But there is a much better way—one that is simpler and more precise.

⁴⁶

That sounds exciting.

What is this new way?

That's what we'll discover in the next chapter, where we also find out how to revise θ₁!

⁴⁷

What's the best snack to help prepare for that next chapter?

¹

That's the way it crumbles.

Let's roll!

We start with a picture

²

This graph looks different, what is it?

It is a graph of the loss in our example against θ₀ while keeping θ₁ at 0.0. The y-axis here represents the loss, and the x-axis represents θ₀, which we also refer to as weight.

³

So, for any possible value of the weight, this graph shows the corresponding value of the loss.

What are the big orange dots?

To draw this picture, we choose five weights −1.0, 0.0, 1.0, 2.0, and 3.0.

For each weight, we determine its corresponding loss, while keeping θ₁ at and using the data set

(line-xs, line-ys)

The orange dots represent the losses at each of these weights.

⁴

How can we find these five losses?

First, let's take a deeper look at our objective function

((l2-loss line) line-xs line-ys)

Show a same-as chart that expands it.

⁵

Here it is

Great.

Let us name this λ-expression obj, for objective function, from frame 64:26.

Explain how we can use obj to determine the losses.

⁶

The λ-expression obj takes the parameters of a line as its argument and results in a scalar representing how closely that line fits the data set (i.e., the loss).

So we must construct a θ with each of those five weights as θ₀ and 0.0 as θ₁. The corresponding losses would be

• (obj (list −1.0 0.0)), which is 126.21
• (obj (list 0.0 0.0)), which is 33.21
• (obj (list 1.0 0.0)), which is 0.21
• (obj (list 2.0 0.0)), which is 27.21
• (obj (list 3.0 0.0)), which is 114.21

Correct.

We use this orange style of graph to show quantities that are not part of our data set.

What is our initial estimate for θ₀?

⁷

It is 0.0.

The loss when θ₀ is 0.0 (and keeping θ₁ at 0.0) is 33.21 from frame 65:29.

The point (0.0, 33.21) is circled in this graph

⁸

Aha!

So the loss seems to be the lowest at the bottom of this curve.^†

Correct.

We need to “roll” down this incline to get to its bottom, and from frame 67:37, we have discovered that we can use the rate of change to roll down faster than we could before without overshooting.

⁹

How can we represent rates of change on this graph?

Here is how we do it for our initial estimate with θ₀ being 0.0

¹⁰

What is the turquoise colored line?

That turquoise line is different from the line for which we are trying to find θ₀ and θ₁. This line is known as a tangent.

A tangent touches the loss curve at exactly one point.

¹¹

Is that point (0.0, 33.21)?

Yes, it is.

The rate of change that we have determined is the

slope of the tangent

This slope has a different name so we don't confuse it with θ₀. It is known as the

gradient

¹²

Could we see the tangent for our next θ₀ at 0.6623?

Here are the tangents for our first two estimates of θ₀

Describe what's interesting about this graph.

¹³

We see that the tangent for our second θ₀ is less steep than the tangent for our first θ₀. Thus, the gradient of the second is lower than the gradient of the first.

These are only for θ₀, but we must also have them for θ₁.

Is there something similar for θ₁?

Well done.

A gradient is a general way of understanding the rate of change of a parameterized function with respect to all its parameters.

¹⁴

How can we find this gradient?

To find the gradient of a function at given values of its arguments, we need to use a new function ∇. The first argument to ∇ is a function f that takes a list of tensors, including θs. The second argument to ∇ is the list for which we need gradients of f.

¹⁵

Does this function then result in the gradients?

Yes.^†

The result of ∇ is a list of gradients of f with respect to each parameter in θ, and is referred to as the gradient list.

So, for example, if we want the gradient of f with respect to parameters u and v, we write it

(∇ f (list u v))

¹⁶

How about an example?

That's a great idea.

Let's find the gradient of sqr with respect to its argument at 27.0

1. 1.  (∇ (λ (θ) (sqr θ₀)) (list 27.0))
2. 2.  (list 54.0)

The first argument to ∇ is a function that expects a θ containing a single parameter, and it squares that single tensor using sqr. The second argument is the θ containing the parameter 27.0. This expresses the gradient of sqr at the scalar 27.0.^†

¹⁷

Instead of these lists, couldn't we directly use scalars here?

A great question.

While using scalars directly without packaging them in a list would make sense for simple examples, more complex learning problems require a very large collection of parameters.

It is easiest to package them in a list, so we have designed ∇ to work exclusively with lists.

¹⁸

So, does ∇ always accept a function and a list of parameters and result in a list of gradients, one for each parameter?

Yes, that is correct.

∇ always results in a list of gradients—one for each parameter provided in θ.

¹⁹

How can we use ∇ to determine the gradient of l2-loss with respect to

θ₀ and θ₁

where each parameter starts out as

0.0?

Here is the gradient of obj, which is the objective function

((l2-loss line) line-xs line-ys)

at θ₀ and θ₁ equal to 0.0

²⁰

So the gradient of obj

with respect to θ₀ is −63.0

and the gradient of obj

with respect to θ₁ is −21.0

But why is this value

−63.0

different from the rate of change

−62.63

of

θ₀

from frame 66:34?

An excellent question.

Our way of finding the rate of change in frame 66:34 is an approximation.

∇ yields more precise results and also yields the gradient with respect to θ₁ at the same time.

²¹

Now that we have a way to find the rate of change, should we use it repeatedly as we did in frame 69:43 to find the well-fitted θ for obj?

Indeed! We revise our θ using ∇ a little bit at a time, over many revisions.

To do that, let's define an iteration function that makes things easier for us.

²²

Does an iteration function help us repeat a function invocation over and over again?

It does.

Here is a skeleton of a function revise that takes a revision function f, which does not change, a natural number revs, and an accumulator θ. It revises θ revs times, each time yielding a new value for θ by invoking f on the current value of θ

Here, revs is the number of revisions of θ remaining and θ is the value accumulated so far. When revs reaches 0, it results in the accumulated θ. When revs has not counted down to 0, we have R, which must revise θ once.

Find R.

²³

R should recursively invoke revise with the new values of revs and θ. We need to provide a new θ by invoking the revision function f on θ

(f θ)

Similarly, since we have finished a revision, the new value of revs must be

(sub1 revs)

Thus, R must be

(revise f (sub1 revs) (f θ))

Here's revise

²⁴

Can we try it out?

Sure.

Using revise, write a same-as chart with this revision function f

(λ (θ)

  (map (λ (p)

    (− p 3))

   θ))

this starting revs

5

and this initial θ

(list 1 2 3)

The final θ is

(list −14 −13 −12)

²⁵

What is map?

Well-timed question!

In its most specific form map accepts a function and a list, and it invokes the function on every member of that list, to produce a new list that is the result of map. For example

1. 1.  (map (λ (x)
2.        (add1 x))
3.       (list 5 7 3))
4. 2.  (list (add1 5) (add1 7) (add1 3))
5. 3.  (list 6 8 4)

²⁶

If this is the most specific form, is there a more general form of map?

In the more general form map accepts more than one list and invokes its function on corresponding members of each list

1. 1.  (map (λ (x y)
2.        (+ x y))
3.       (list 12 17 32)
4.       (list 8 3 11))
5. 2.  (list (+ 12 8) (+ 17 3) (+ 32 11))
6. 3.  (list 20 20 43)

²⁷

So, in frame 25, invoking map on the revision function f and θ produces a new list where every member is 3 less than the corresponding member of the θ.

Here's the start of the same-as chart

1. 1.  (revise f 5 (list 1 2 3))
2. 2.  (revise f 4 (f (list 1 2 3)))
3. 3.  (revise f 4 (list −2 −1 0))
4. 4.  (revise f 3 (f (list −2 −1 0)))
5. 5.  (revise f 3 (list −5 −4 −3))

Now complete the same-as chart.

²⁸

Thanks for the hint

1.  6.  (revise f 2 (f (list −5 −4 −3)))
2.  7.  (revise f 2 (list −8 −7 −6))
3.  8.  (revise f 1 (f (list −8 −7 −6)))
4.  9.  (revise f 1 (list −11 −10 −9))
5. 10.  (revise f 0 (list −14 −13 −12))
6. 11.  (list −14 −13 −12)

Now let's get back to defining a function that guides us from one θ to the next.

The third argument to revise must be a list containing the initial values of θ₀ and θ₁, in that order.

What would this initial θ look like?

²⁹

Since we're initializing θ₀ and θ₁ at 0.0, the initial θ should be

(list 0.0 0.0)

Could we see what f , this revision function, looks like?

Sure.

Here is a skeleton for this, where revs is 1000 and the learning rate α is 0.01

Explain how the invocation of revise in this example works.

³⁰

We carry out 1000 revisions, starting with (list 0.0 0.0) and invoking f for each revision.

The revision function accepts an initial θ and then produces a different θ, and repeats this for 1000 revisions.

Very good.

In the revision function, we determine the gradient of the objective function with respect to θ₀ and θ₁, which are packaged into a θ.

We use ∇ to get the gradient list gs, which is a list of two scalars.

The first is the

gradient of the loss with respect to θ₀

and the second is the

gradient of the loss with respect to θ₁

Now find the expressions for W and B.

³¹

Since the revision function must produce a new θ

W should be the new θ₀

and

B should be the new θ₁

Correct.

And how do we determine these new members of θ i.e., the new parameters?

³²

We multiply the gradient for a given parameter by the learning rate α and subtract it from that parameter.

The gradient with respect to θ₀ is given by the 0th member of the gradient list gs, which is gs₀. So the expression W is

(− θ₀ (* α gs₀))

Similarly, because the gradient with respect to θ₁ is given by gs₁, the expression B is

(− θ₁ (* α gs₁))

Yes, well done.

Here is the completed revision function f with W and B filled in. We show the invocation of revise using this new f in a revision chart.

We use revision charts to show the results after multiple revisions

³³

So, this means that after 1000 revisions, we get a reasonably well-fitted θ.

What does this line

(list 1.05 1.87e−06)

look like?

³⁴

That's exciting! It is almost identical to the visually estimated line from frame 25:20.

Another question, though. Why do we have revs at 1000? Are we certain we'll reach a well-fitted θ by then?

A very good question.

We pick a high enough number for revs that we know gets us close enough to the well-fitted θ.

Depending upon the kind of function we are optimizing, we can pick an appropriate revs (usually through a combination of the size of the data set and experimentation).

³⁵

That sounds a little ad hoc. Is there a better approach?

Yes, but for only certain kinds of problems.

As we encounter problems with θs that have much larger tensors and target functions that do much more interesting things, a fixed number for revs is usually a good approach.

³⁶

Does this way of finding θ₀ and θ₁ have a name?

Yes, it does.

This algorithm is known as

optimization by gradient descent ^†

³⁷

Who knew sliding down a slippery slope would be this much fun!

It sure is!

There is one more simplification we can make to our revision function

f

Let's look at the expression in our revision function that produces a new θ

(list

  (− θ₀ (* α gs₀))

  (− θ₁ (* α gs₁)))

³⁸

What about it?

This expression is specific to a θ that contains exactly two parameters.

³⁹

Oh, so it works for target functions (like line) that need exactly two parameters in their θs.

Exactly.

In general, we must produce a new θ independent of the length of θ, and correspondingly of the length of gs.

⁴⁰

Can we use map here?

Yes, we can.

At each member of θ and gs, we have to subtract α times that member of gs from a member of θ. We can write this as a function of two arguments where p is a member in θ and g is the corresponding member in gs

(λ (p g)

  (− p (* α g)))

⁴¹

Oh, now we map this function over

θ

and

gs

Correct.

We now rewrite the invocation of revise

(let ((f (λ (θ)

     (let ((gs (∇ obj θ)))

      (map (λ (p g)

         (− p (* α g)))

       θ

       gs)))))

  (revise f 1000 (list 0.0 0.0)))

⁴²

So we have replaced the lines

(list

  (− θ₀ (* α gs₀))

  (− θ₁ (* α gs₁)))

with a similar invocation using map from frame 25

(map (λ (p g)

   (− p (* α g)))

  θ

  gs)

Yes, correct.

The parameterized function given by

((l2-loss line) line-xs line-ys)

is the objective function from frame 64:26 because our objective is to find the θ that minimizes this function, i.e., that rolls us down the incline to the lowest point, representing the lowest loss.

We now express the complete gradient descent for our data set

(let ((α 0.01)

   (obj ((l2-loss line) line-xs line-ys)))

  (let ((f (λ (θ)

     (map (λ (p g)

       (− p (* α g)))

      θ

      (∇ obj θ)))))

   (revise f 1000 (list 0.0 0.0))))

⁴³

Okay.

We are going to make a temporary adjustment. Except for the initial value of θ, we have two “constant scalars” in this expression

The number of revisions revs

and

the learning rate α

For now, let's define these names

How should we now express gradient descent?

⁴⁴

Like this?

(let ((obj ((l2-loss line) line-xs line-ys)))

  (let ((f (λ (θ)

   (map (λ (p g)

     (− p (* α g)))

    θ

    (∇ obj θ)))))

  (revise f revs (list 0.0 0.0))))

Excellent!

In this expression, the value of obj

((l2-loss line) line-xs line-ys)

and the initial value of θ

(list 0.0 0.0)

become arguments to a function named

gradient-descent

⁴⁵

Okay.

We now define this algorithm as the function gradient-descent. The Θ^† is a renaming of the formal of f from page xxiii

This function gradient-descent gives us the ability to find the well-fitted θs of many different objective functions with their own θs.

⁴⁶

And it is what we started out looking for!

What is Θ here?

The Θ is a formal of f , the revision function.^†

⁴⁷

Why have we introduced Θ?

Now it seems frivolous, but in fact we have

grate expectations^†

for

Θ

But for now, Θ is simply a name we use for the formal of the revision function.

⁴⁸

Okay, for now.

Where can the initial θ and Θ be found?

⁴⁹

θ appears only where we invoke revise.

Θ appears only in the revision function.

Here is how we use a revision chart with gradient-descent to learn θ₀ and θ₁ with their initial estimates at 0.0

⁵⁰

It's the same as the result in frame 33.

We now have a new learning toy!

Yes, we do!

We'll play with it in the upcoming interlude.

⁵¹

Whew! That's a relief.

¹

Could not get enough of it.

This interlude is about hyperparameters.

²

What are hyperparameters?

The temporarily-defined names like revs and α that control the behavior of gradient-descent are known as hyperparameters.

Hyperparameters are always names associated with scalars like these.

³

Why are they important?

These scalars vary by the problem under consideration and must be selected after some thought and experimentation.

⁴

So the scalars that we have picked for α and revs in gradient-descent from chapter 4 may not necessarily work in other situations?

That is correct.

There are other hyperparameters we encounter later.

To make our lives easier when using them, we introduce a new construct.

⁵

Interesting.

Can we pack them all up into a list as we did for θ?

No, a better way would be to treat them as a special kind of name.

⁶

What makes them special?

Once these names are declared as hyperparameters, they are available to be used in any function.

⁷

Okay.

Here is a nonsensical example

Now, what is

(+ smaller larger)?

⁸

Because neither smaller nor larger is associated with a scalar, it has no value.

⁹

That should be 2001.^†

Will either smaller or larger still be a scalar outside with-hypers?

No, they won't.

These hyperparameters have scalars associated with them only during with-hypers expressions.

¹⁰

Is with-hypers similar to let?

No, it is not.

A let-expression creates a new local name for a value, but a with-hypers provides a new value for a pre-existing name that has been declared as a hyperparameter.

¹¹

That clarifies it!

Once these hyperparameters have scalars associated with them by with-hypers, those scalars are available to all functions using them.

After the body

(+ smaller larger)

if with-hypers has yielded a result, the hyperparameters would no longer have values.^†

¹²

So the hyperparameters are like our defined names, but they can be associated with scalars only when set up using with-hypers?

Yes, correct.

Now it gets more interesting. Suppose we define nonsense?, this mostly useless function

What is (nonsense? 6)?

¹³

Again, it has no value for the very same reason as before. The hyperparameter smaller has not yet been provided a scalar.

We discover, by providing a scalar for the hyperparameters in use

(with-hypers

  ((smaller 5))

  (nonsense? 6))

that the result is #t. To see how this works, imagine that the definition of nonsense? is also inside the with-hypers, and work it out from there.

¹⁴

Is (nonsense? 6) the body of this with-hypers?

Yes!

Explain how we get this result.

¹⁵

When nonsense? is invoked within with-hypers, the scalar smaller is 5, so (nonsense? 6) results in #t.

So why is nonsense? dashed?

That's because it is silly to use the name

nonsense?

¹⁶

Are we going to use hyperparameters in a new

gradient-descent?

Indeed, but that is for the next chapter.

¹⁷

Can't wait!

¹

Can't wait!

It's time to generalize what we have learned so far.

²

Is this where we find out how Θ meets our grate expectations?

No, we are not there yet, but we are getting closer.

³

Where should we start?

Now, we declare two sensible hyperparameters

We have seen revs and α before, but not as hyperparameters.

⁴

How do we use these hyperparameters for gradient-descent?

Here's the dashed gradient-descent from frame 89:46. It repeats a single revision revs number of times, and for each revision it refines the result of the previous revision

Now, remember l2-loss?

⁵

Here is l2-loss from frame 63:22

(define l2-loss

  (λ (target)

   (λ (xs ys)

    (λ (θ)

     (let ((pred-ys ((target xs) θ)))

      (sum

       (sqr

        (− ys pred-ys))))))))

This definition of gradient-descent uses the two hyperparameters

revs

and

α

Using l2-loss and with-hypers from frame 94:9, provide a scalar for each hyperparameter and show how to invoke gradient-descent in a revision chart.

⁶

Here is the revision chart

We have gotten the same result from frame 90:49, and we have cleanly separated out our hyperparameters.

The definitions of gradient-descent and l2-loss in frame 5 take functions as arguments and are not tied to a particular target function (like line).

⁷

This means that we can use these two functions for learning the θ for arbitrary target functions.

That is liberating!

Here is a new data set

Draw its corresponding graph.

⁸

These points are not on a line

Indeed, they are not. So, we must learn the θ of a different target function.

For this example, we use a simple nonlinear function.

⁹

What is a nonlinear function?

A nonlinear function is not linear in its arguments.

¹⁰

And …

A linear function is one that uses only addition and scaling to find its result.

¹¹

What is scaling?

Scaling multiplies its argument by a fixed value or by a parameter. For example

(* 5.0 x)

scales the value of x by 5.0.

And if we have a parameter

θ₀

then

(* θ₀ x)

scales the value of x by θ₀.

¹²

So, this is why line is a linear function. It scales its argument x with θ₀ and adds θ₁ to it.

What nonlinear function does the data set in frame 8 represent?

A very good question.

Let's make the assumption that this data set is quadratic.

¹³

What does quadratic mean?

It means that it can be predicted by a quadratic function.

¹⁴

What is a quadratic function?

Here is a general definition of a quadratic function^†

The function quad has three scalar parameters provided in a θ.

¹⁵

Could we see an example of how quad works?

Sure.

Let's take

t to be 3.0

and

θ to be (list 4.5 2.1 7.8)

Show a same-as chart for this example.

¹⁶

Here it is

Does the use of sqr make quad nonlinear?

Indeed it does.

Now we use gradient-descent with l2-loss as the loss function to learn the θ for this data set with quad as the target function.

And what is the new expectant function from frame 63:24?

¹⁷

The expectant function is the loss function invoked on the target

(l2-loss quad)

And what is the objective function?

¹⁸

The objective function from frame 64:26 is the function obtained by invoking the expectant function on the new data set

((l2-loss quad) quad-xs quad-ys)

Great.

We invoke gradient-descent on this objective function and an initial θ.